feat: committor service #366

thlorenz · 2025-05-15T09:19:50Z

Summary

Adding a full fledged committor service to support the following kind of commits:

commits up to 10Kb as long as the size increase per commit is <=1Kb
max accounts per bundle is 3 without lookup tables and 12 with lookup tables
commit results are persisted into a SQlite database in the ledger directory which makes it
easy to manually retry them or do this automatically (next committor version will support this)

Details

The following crates were developed
here and integrated
into the validator:

magicblock-rpc-client

thin wrapper around the solana rpc client with added features like improved transaction
signature status checking (including sane retries)
lookup table convenience methods
multi account fetch with known max limit
waiting for slots

magicblock-table-mania

a lookup table manager that allows to create, extend and deactivate + close lookup tables
the API supports reserving pubkeys which will be added to a table and when they need to be
used in a transaction the needed table addresses are provided
once a pubkey is released this is noted and a table is deactivated and closed on chain once
all its pubkeys are released

magicblock-committor-program

allows the committor service to upload account data to be committed in chunks
supports creation/resizing of buffers and filling them with transactions running in parallel
(out of order is ok)

magicblock-committor-service

commits a changeset as fast and efficiently as possible
first prefers transactions that don't require lookup tables since those take time to become
available
then prefers args only commits to the use of buffers (which require preparation)

Sample Flow

Validator starts up and reserves common pubkeys (used in all commits) with the committor
service
Account are cloned and for each account the pubkeys for commits are reserved to prepare the
lookup tables which we might need as fast as possible
Account(s) commit is scheduled and registered as we did before
The commits are processed via the committor service and the on chain transaction signature
is provided to the user via logs in a transaction (as we did before, just now we have to wait
for the commit to complete)

For those commits the committor service ensures that accounts with the same commit (bundle) id
are always processed atomically.

The committor service also picks the best possible strategy to commit each changeset,
preferring speed and then cost.

It also inserts configurable (via code) compute budget instructions to each transaction.

The outcome of each commit is persisted to a database which allows manual (and in the next
version) automated retries of failed commits.
Succeessful commits are also persisted to the database and can be used for diagnostics. In the
future they should be removed since no retry is necessary.

On chain signatures for process-commit/finaliz/undelegate transactions are also persisted to
the database in a separate table.

This table is queried by the validator to log those signatures as part of a transaction that
the user waits for.

Greptile Summary

This PR introduces a comprehensive committor service to the MagicBlock validator for managing account changes and transactions. The implementation includes a new Solana program for handling commits, a service layer for managing commit operations, and integration with lookup tables for transaction optimization.

Added magicblock-committor-program that handles chunked account data commits with proper buffer management and security checks
Added magicblock-committor-service that orchestrates commit operations with support for bundling, lookup tables, and persistent status tracking via SQLite
Added magicblock-table-mania for managing address lookup tables to optimize transaction sizes
Added magicblock-rpc-client with enhanced transaction handling and batched account fetching
Integrated committor service with existing account management systems while maintaining backward compatibility

- at this point schedule commit tests pass with maximum concurrency

- https://github.com/magicblock-labs/committor/tree/73f6a93d43995991fcc684b8853041f43fc19fa2

…m setup

* master: release: 0.1.2 (#359) Bmuddha/fix/limit num monitoring accounts (#356)

greptile-apps

_{113 file(s) reviewed, 128 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

magicblock-account-cloner/src/account_cloner.rs

magicblock-account-cloner/src/remote_account_cloner_worker.rs

magicblock-accounts/Cargo.toml

test-integration/test-runner/bin/run_tests.rs

test-integration/test-table-mania/tests/ix_lookup_table.rs

test-integration/test-table-mania/tests/ix_reserve_pubkeys.rs

test-integration/test-tools/src/validator.rs

This reverts commit 776bbf1.

…ing issues" This reverts commit aeeb5ae.

bmuddha

Good job at documenting the code so thoroughly, left some nits, I have to trust that the logic is sane and the test suite covers all the edge case, as trying to completely understand the entire interplay of all the components would take prohibitive amount of time.

bmuddha · 2025-05-18T17:07:15Z

magicblock-account-cloner/src/account_cloner.rs

+    res.await
+        .map_err(|err| {
+            // Send request error
+            AccountClonerError::CommittorSerivceError(format!("{:?}", err))


Suggested change

AccountClonerError::CommittorSerivceError(format!("{:?}", err))

AccountClonerError::CommittorSerivceError(format!("error sending request {err:?}"))

bmuddha · 2025-05-18T17:07:26Z

magicblock-account-cloner/src/account_cloner.rs

+    res.await
+        .map_err(|err| {
+            // Send request error
+            AccountClonerError::CommittorSerivceError(format!("{:?}", err))


Suggested change

AccountClonerError::CommittorSerivceError(format!("{:?}", err))

AccountClonerError::CommittorSerivceError(format!("error sending request {err:?}"))

```suggestion

AccountClonerError::CommittorSerivceError(format!("{:?}", err))

bmuddha · 2025-05-18T17:07:47Z

magicblock-account-cloner/src/account_cloner.rs

+        })?
+        .map_err(|err| {
+            // Commit error
+            AccountClonerError::CommittorSerivceError(format!("{:?}", err))


Suggested change

AccountClonerError::CommittorSerivceError(format!("{:?}", err))

AccountClonerError::CommittorSerivceError(format!("error during commit {err:?}"))

bmuddha · 2025-05-19T12:52:40Z

magicblock-accounts/src/remote_scheduled_commits_processor.rs

+        let max_slot = scheduled_commits
+            .iter()
+            .map(|commit| commit.slot)
+            .max()
+            .unwrap();
+        // Safety we just obtained the max slot from the scheduled commits
+        let ephemeral_blockhash = scheduled_commits
+            .iter()
+            .find(|commit| commit.slot == max_slot)
+            .map(|commit| commit.blockhash)
+            .unwrap();

+        changeset.slot = max_slot;


Suggested change

let max_slot = scheduled_commits

.iter()

.map(|commit| commit.slot)

.max()

.unwrap();

// Safety we just obtained the max slot from the scheduled commits

let ephemeral_blockhash = scheduled_commits

.iter()

.find(|commit| commit.slot == max_slot)

.map(|commit| commit.blockhash)

.unwrap();

changeset.slot = max_slot;

let mut changeset = Changeset::default();

let Some(commit) = scheduled_commits.iter().max_by_key(|commit| commit.slot) else {

return Ok(());

};

let max_slot = commit.slot;

let ephemeral_blockhash = commit.blockhash;

changeset.slot = max_slot;

bmuddha · 2025-05-20T06:51:31Z

magicblock-committor-program/src/error.rs

+        msg!("Error: {:?}", e);
+        use CommittorError::*;
+        let n = match e {
+            UnableToSerializeChangeSet(_) => 0x69000,


What are those magic numbers? Some explanatory comments would be helpful

bmuddha · 2025-05-20T15:31:41Z

magicblock-table-mania/src/lookup_table_rc.rs

+    /// - *returns* `true` if the pubkey could be reserved
+    fn reserve(&self, pubkey: &Pubkey) -> bool {
+        if let Some(count) = self.pubkeys.get(pubkey) {
+            count.fetch_add(1, Ordering::SeqCst);


SeqCst is a total overkill, Relaxed would be totally correct as we are not synchronizing any other state through this atomic

bmuddha · 2025-05-20T15:34:37Z

magicblock-table-mania/src/lookup_table_rc.rs

+    fn release(&self, pubkey: &Pubkey) -> bool {
+        if let Some(count) = self.pubkeys.get(pubkey) {
+            count
+                .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |x| {


ditto, SeqCst is an overkill, since no other state is synchronized

bmuddha · 2025-05-20T15:35:04Z

magicblock-table-mania/src/lookup_table_rc.rs

+    fn has_reservations(&self) -> bool {
+        self.pubkeys
+            .values()
+            .any(|rc_pubkey| rc_pubkey.load(Ordering::SeqCst) > 0)


ditto, Ordering::Relaxed is enough

bmuddha · 2025-05-20T15:37:13Z

magicblock-table-mania/src/lookup_table_rc.rs

+        })
+    }
+
+    /// Returns `true` if the we requested to deactivate this table.


Suggested change

/// Returns `true` if the we requested to deactivate this table.

/// Returns `true` if we have already requested to deactivate this table.

bmuddha · 2025-05-20T15:51:27Z

magicblock-table-mania/src/manager.rs

+
+#[derive(Clone)]
+pub struct TableMania {
+    pub active_tables: Arc<RwLock<Vec<LookupTableRc>>>,


I guess some research was made to justify the usage of async synchronization primitives, i.e. they will be used in highly concurrent environments and with lock guards being held across await points, if not, then their use is unjustified.

thlorenz added 30 commits May 7, 2025 11:10

test: add realloc ix to flexi counter

45bdf72

chore: resort workspace depencencies

2e61882

chore: retire old remote scheduled commits processor and add new one

6806acf

feat: initial impl of commits processor based on committor service

8cdc141

feat: initializing committor service at startup

c0d08a4

test: logging signature of failed init_committees tx

9eed608

test: adapt to expect two signatures when finalizing

dae8ea8

feat: ensure ephemeral validator is funded on chain

80e6777

test: add single account commit test

3078a1c

test: warn when we have issues fetching a transaction

8391124

fix: adding change for validator fund check

e5aa347

feat: fully integrating committor service

be98740

chore: update ix tests cargo

5ce7a20

chore: comment with requirements for schedule commit tests

b54cd3b

chore: improve test logs

dc2da58

chore: include compute unit price when initializing committor

560fd6b

fix: mark accounts to be undelegated

b3445c1

chore: properly handle some unwraps

24bb27a

test: general improvements + fixes

cc389e7

- at this point schedule commit tests pass with maximum concurrency

chore: use trait for committor service and create stub to use in tests

f7304cf

chore: update ensure accounts tests to use stub

04b0116

chore: re-enable account cloner and ensure accounts tests

d770b95

chore: move committor repo crates into magicblock monorepo

88dd40b

- https://github.com/magicblock-labs/committor/tree/73f6a93d43995991fcc684b8853041f43fc19fa2

chore: move committor service integration tests

e6f1edd

chore: noting escrow/fee payer related test requirements

1c4dd78

chore: minor cleanup in test runner

8c620fe

ix: load committor program for schedule commits

f3e17d1

ix: allow configuring loaded accounts

6d109f9

ix: match ephem validator keypair to the one we use on chain

0dceb8b

ix: ensure we always pass same loaded chain accounts to chain + ephen…

7fdc795

…m setup

thlorenz added 2 commits May 15, 2025 15:17

Merge branch 'master' into thlorenz/committor

d3bacca

* master: release: 0.1.2 (#359) Bmuddha/fix/limit num monitoring accounts (#356)

chore: fmt

7bde596

greptile-apps bot reviewed May 15, 2025

View reviewed changes

thlorenz added 17 commits May 15, 2025 16:48

chore: update delegation program reference

52fd831

chore: opt out of doctests for added crates

714dd68

ix: add rule to make committor program

0f94802

fix: error misspelling

c8ddb16

chore: address some greptiles

b246170

ix: check in missing config

d3892d1

chore: more greptiles

ca5e9f4

ix: move table mania/committor tests last since they are the slowest

bc48d58

chore: rollback delegation program version

9fb082a

chore: cleanup stray log

ee396da

ix: give more compile time before expecting validator to listen

8635738

chore: greptiles

7f3e0eb

chore: improved error handling in table mania manager

28504c1

fix: greptiles

bc1324d

chore: demote some no longer urgent TODOs

644ad1c

feat: committor service persists into ledger path

4f8ecc8

chore: remove duplicate code in magic validator

3cf473d

thlorenz requested review from GabrielePicco, bmuddha and taco-paco May 16, 2025 04:28

thlorenz added 6 commits May 16, 2025 10:30

feat: limiting stale reallocs until we bail

a7b4aa5

tmp: disabling ledger restore tests to isolate issues

776bbf1

tmp: disable all workflows but integration tests while isolating issues

aeeb5ae

fix: ordering of worker startups

aedf9a2

Revert "tmp: disabling ledger restore tests to isolate issues"

1e9921d

This reverts commit 776bbf1.

Revert "tmp: disable all workflows but integration tests while isolat…

0e0ef0f

…ing issues" This reverts commit aeeb5ae.

bmuddha approved these changes May 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: committor service #366

feat: committor service #366

thlorenz commented May 15, 2025 •

edited

Loading

greptile-apps bot left a comment

bmuddha left a comment

bmuddha May 18, 2025

bmuddha May 18, 2025

bmuddha May 18, 2025

bmuddha May 19, 2025

bmuddha May 20, 2025

bmuddha May 20, 2025

bmuddha May 20, 2025

bmuddha May 20, 2025

bmuddha May 20, 2025

bmuddha May 20, 2025

	AccountClonerError::CommittorSerivceError(format!("{:?}", err))
	AccountClonerError::CommittorSerivceError(format!("error sending request {err:?}"))

	/// Returns `true` if the we requested to deactivate this table.
	/// Returns `true` if we have already requested to deactivate this table.

feat: committor service #366

Are you sure you want to change the base?

feat: committor service #366

Conversation

thlorenz commented May 15, 2025 • edited Loading

Summary

Details

magicblock-rpc-client

magicblock-table-mania

magicblock-committor-program

magicblock-committor-service

Sample Flow

Greptile Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

bmuddha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thlorenz commented May 15, 2025 •

edited

Loading